Methods and systems for determining whether a monitored individual&#39;s hand(s) have entered a virtual safety zone

ABSTRACT

Systems and methods are provided that allow caregivers, central monitoring services, and other persons to monitor whether a monitored individual&#39;s hands have entered into an area in which the hands are prohibited, such as areas in which the monitored individual may remove or disturb a piece of medical equipment. When the monitored individual&#39;s hands do enter the restricted area that is represented by an electronic virtual safety zone, an alert is generated and transmitted to a designated recipient.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of and claims priority to U.S.Nonprovisional application Ser. No. 14/757,593, titled “Method andSystem for Determining Whether a Monitored Individual's Hand(s) HaveEntered a Virtual Safety Zone” filed Dec. 23, 2015, which claims thebenefit of U.S. Provisional Application No. 62/096,289, filed on Dec.23, 2014, both of which are herein incorporated by reference in theirentireties.

TECHNICAL FIELD

The present disclosure relates to methods and systems for determiningwhether a monitored individual's hand or hands have entered a virtualsafety zone.

BACKGROUND

Traditionally, monitoring of hospital patients is a costly,time-consuming endeavor. Of great concern to caregivers is a patienttouching, removing, or otherwise disturbing medical equipment. If apatient disturbs an IV tube, breathing tube, catheter, or other medicalequipment, significant negative repercussions can follow. These includehealthcare-associated infections (HAIs), which are infections acquiredby patients during the course of receiving treatment for other healthconditions. According to recent studies, one in every twentyhospitalized patients will acquire an infection during the course ofreceiving healthcare treatment for a different condition, In terms ofthe economic impact, studies estimate the overall annual direct medicalcosts of HAIs range between $28.4 and $45 billion. The medical facilitymust typically bear the cost of the HAI, which puts a strain on thefinances of the healthcare provider. In addition to HAIs, other seriousinjuries or even death can result from a patient's improper removal ordisturbance of a piece of medical equipment.

BRIEF SUMMARY OF THE DISCLOSURE

This brief summary is provided as a general overview of the moredetailed disclosure which follows. It is not intended to identify key oressential elements of the disclosure, or to define the claim terms inisolation from the remainder of the disclosure, including the drawings.

In general, this disclosure relates to systems, methods, andcomputer-readable storage media that notify caregivers or other monitorsif a patient's hand(s) have entered into an area where the caregiver ormonitor has determined that the patient's hand(s) should not be. Forexample, regardless of the patient's intent, it may be undesirable forpatients to touch or manipulate certain medical equipment or temporaryimplants, such as nasogastric tubes, tracheal tubes, central lines,Intravenous (IV) lines, and the like. Simply touching the equipment maycontaminate and/or disturb it. Further, disoriented and/or uncomfortablepatients may attempt to reposition and/or remove medical equipment ortemporary implants, which can cause injury—including possibly severeinjury—to the patient. If the patient's activity involves the removal ofsharp and/or body-fluid contaminated equipment, the patient's activitymay also pose a risk to caregivers or others who approach the patient(e.g., because they could prick themselves and/or be exposed to blood-or other body fluid-borne pathogens). However, continuous visualmonitoring of a patient is often impractical. Even in home careenvironments or other settings where the caregiver to patient ratio maybe 1:1, there will be moments when a caregiver needs to tend tasks thatmay take them out of visual range of the patient.

In some aspects, a method is disclosed for detecting when a monitoredindividual has moved one or both of his or her hands within a virtualsafety zone. The method may comprise configuring a virtual safety zonearound an area where the individual to be monitored is located. Themethod may comprise providing one or more 3D motion sensors to capturelive video data from the area. The method may comprise forwarding thevideo data from the one or more 3D motion sensors to a computerizedmonitoring system. The method may comprise determining, as by thecomputerized monitoring system, when the monitored individual has movedat least one of his or her hands within the virtual safety zone. Themethod may comprise electronically transmitting an alert to acentralized monitoring system, by the computerized monitoring system,when the computerized monitoring system determines that the monitoredindividual has moved at least one of his or her hands within the virtualsafety zone.

The method may comprise determining whether the at least one hand of themonitored individual remained within the virtual safety zone for apredetermined period of time before electronically transmitting an alertto the centralized monitoring system. The method may comprisecontinuously displaying a live video feed of the monitored area receivedfrom the one or more 3D motion sensors on a centralized monitoring alertdisplay after it has been determined that at least one hand of themonitored individual is within the virtual safety zone for thepredetermined period of time. The method may comprise continuouslydisplaying a live video feed of the monitored area received form the oneor more 3D motion sensors on a centralized monitoring primary displaythat is remotely located from the monitored area. The method maycomprise continuously displaying a live feed of the monitored areareceived form the one or more 3D motion sensors on a centralizedmonitoring alert display after it has been determined that at least onehand of the monitored individual is within the virtual safety zone,wherein the centralized monitoring alert display is a separate displayfrom the centralized monitoring primary display.

The method may comprise continuously displaying a live video feed of themonitored area received from the one or more 3D motion sensors after ithas been determined that at least one hand of the monitored individualis within the virtual safety zone. The method may comprise updating adatabase in communication with the computerized monitoring systemregarding the determination that at least one hand of the monitoredindividual was within the virtual safety zone. The method may comprisenotifying a designated caregiver by electronic message regarding thedetermination that at least one hand of the monitored individual waswithin the virtual safety zone. The virtual safety zone may encompass atleast part of the monitored individual's face. Determining, by thecomputerized monitoring system, when the monitored individual has movedat least one of his or her hands within the virtual safety zone maycomprise using facial tracking to monitor the virtual safety zone.Determining, by the computerized monitoring system, when the monitoredindividual has moved at least one of his or her hands within the virtualsafety zone may comprise determining that the face of the monitoredindividual is at least partially obscured.

In some aspects, the disclosure relates to a system for determiningwhether a monitored individual has placed one or both hands in a virtualsafety zone. The system may comprise one or more 3D motion sensorsco-located with a monitored individual. The system may comprise acomputerized monitoring system. The computerized monitoring system maybe configured to receive data from the one or more 3D motion sensors.The computerized monitoring system may be configured to identify theposition of the monitored individual's hands. The computerizedmonitoring system may be configured to determine whether at least one ofthe monitored individual's hands have entered a virtual safety zone. Thesystem may comprise a computerized communication system. Thecomputerized communication system may be configured to receive from thecomputerized monitoring system a determination that at least one of themonitored individual's hands have entered a virtual safety zone. Thecomputerized communication system may be configured to send an alert ofthe entry into the virtual safety zone to at least one designatedrecipient.

The computerized monitoring system may be further configured to actuatea timer upon determining that the monitored individual's hand or handshave entered a virtual safety zone. The computerized monitoring systemmay send to the computerized communication system a determination thatthe monitored individual's hand or hands have entered a virtual safetyzone only if the monitored individual's hand or hands remain in thevirtual safety zone for at least a predetermined period of time. Thedesignated recipient of the alert may include one or more of acaregiver, the monitored individual, an alternate caregiver, and asupervisor. The alert may comprise an audible instruction to thepatient. The system may further comprise a database for logging eventsrelated to the entry of the monitored individual's hand or hands intothe virtual safety zone. The computerized communication system mayfurther be configured to save log entries for events related to theentry of the monitored individual's hand or hands into the virtualsafety zone and related alerts.

In some aspects, the disclosure relates to computer-readable storagemedia having stored thereon executable instructions. When executed by acomputer, the instructions may cause the computer to receive visual datafrom one or more 3D motion sensors. The instructions may cause thecomputer to identify a monitored individual's location from the visualand/or sound data. The instructions may cause the computer to establisha virtual safety zone at least partially overlapping the monitoredindividual's location. The instructions may cause the computer toidentify the monitored individual's hands from the visual data. Theinstructions may cause the computer to determine whether one or both ofthe monitored individual's hands enter the virtual safety zone from thevisual data. The instructions may cause the computer to time theduration for which the monitored individual's hand or hands remain inthe virtual safety zone. The instructions may cause the computer toalarm if the monitored individual's hand or hands remain in the virtualsafety zone for longer than a predetermined period. The instructions maycause the computer to present a human-readable visual image of themonitored individual's location to a display device. The instructionsmay cause the computer to accept user input to define the virtual safetyzone. The instructions may cause the computer to log alerts in adatabase.

Additional objects, advantages, and novel features of the disclosurewill be set forth in part in the description which follows, and in partwill become apparent to those skilled in the art upon examination of thefollowing, or may be learned by practice of the disclosure.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING

The present disclosure references the attached drawing figures, wherein:

FIG. 1 is a flowchart for an exemplary method for determining whether amonitored individual's hands have entered a virtual safety zone;

FIG. 2 is an exemplary system for centralized monitoring to determinewhether a monitored individual's hands have entered a virtual safetyzone;

FIG. 3 is an exemplary rendering of a composite screen showing threemonitored individuals;

FIG. 4 is an exemplary view of the monitoring image of a monitoredindividual;

FIG. 5 is a rendering of an exemplary virtual safety zone;

FIG. 6 is an exemplary interface for configuring a virtual safety zone;

FIG. 7 is an exemplary display of a monitored individual overlaid with avirtual safety zone;

FIG. 8 is an exemplary interface for configuring options for monitoringan individual;

FIG. 9 is an exemplary interface for configuring a monitoring system;

FIG. 10 is an exemplary interface for configuring a monitoring system;

FIG. 11 is an exemplary interface for configuring a monitoring system;

FIG. 12 is an exemplary interface for configuring a monitoring system;

FIG. 13 is an exemplary interface for configuring a monitoring system;and

FIG. 14 is an exemplary interface for configuring a monitoring system.

DETAILED DESCRIPTION

As mentioned above, patients, particularly but not exclusively patientswho are disoriented or uncomfortable, may resort to self-help andattempt to reposition or remove medical equipment and/or temporaryimplants. When nasogastric, tracheal, central, IV, or other lines orequipment are disturbed by the patient, they may become contaminated bythe patient's hand and contribute to the development of infection.Further, if lines or equipment are removed improperly, the patient caninjure himself or herself. This is sometimes described in terms of apatient, however, the term “monitored individual” is meant to encompassboth “patients” in the sense of individuals under immediate medicalcare, such as patients in an in-patient setting, as well as individualswho may use certain medical equipment and/or temporary implants in othersettings, including, without limitation, assisted living facilities,nursing homes, hospice care, home care, outpatient settings, and thelike.

The monitoring may be done by a caregiver. A caregiver may be a medicalprofessional or paraprofessional, such as an orderly, nurse's aide,nurse, or the like. A caregiver may also be a friend, relative,individual, company, or facility that provides assistance with dailyliving activities and/or medical care for individuals, such asindividuals who are disabled, ill, injured, elderly, or otherwise inneed of temporary or long-term assistance.

FIG. 1 shows an exemplary workflow for monitoring whether a monitoredindividual's hand(s) have encroached into a predefined virtual safetyzone through the use of 3D motion sensors. A 3D motion sensor is anelectronic device that contains one or more cameras capable ofidentifying individual objects, people, and motion regardless oflighting conditions. The 3D motion sensor may further contain one ormore microphones to detect audio. The cameras can utilize technologiesincluding but not limited to color RGB, CMOS sensors, lasers, infraredprojectors, and RF-modulated light. The 3D motion sensor may have one ormore integrated microprocessors and/or image sensors to detect andprocess information both transmitted from and received by the variouscameras. Exemplary 3D motion sensors include the Microsoft® Kinect®Camera, Sony® PlayStation® Camera, and the Intel® RealSense™ Camera,each of which happens to include microphones, although sound capture isnot essential to the practice of the disclosure.

The microprocessor of the 3D motion sensor may be configured tocalculate a change in location of the person or object of interest overa period of time, if a change has occurred. As a non-limiting example, aperson's right hand can be at time T₁ located at coordinates (x₁, y₁,z₁) in a picture frame taken by the camera. At time T₂, the right handis captured by the picture frame taken by the camera at coordinates (x₂,y₂, z₂). Based on this information, motion, speed and direction can bederived utilizing the elapsed time and comparing the two 3D coordinatesover the elapsed time. As opposed to conventional motion sensors, whichuse captured motion to control a camera, the 3D motion and/or soundsensor described herein uses the camera in order to compute the motion.

The 3D motion and/or sound sensors may operate continuously, orintermittently (for example, running for a fixed period at definedintervals, such as periods when the monitored individual might beexpected to be awake, or to be disoriented, or to otherwise meritmonitoring), or on a trigger (e.g., when a motion detector or lightsensor is activated, suggesting activity in the room). Thecamera/sensors are preferably continuously on at all times while themonitoring is occurring, regardless of whether the person or object ofinterest is moving or not. The camera preferably views the entire roomor a large portion of the room simply by its placement in a mannersufficient for the room to be visible to the camera.

The 3D motion sensors may record video. Video is technically made up ofindividual picture frames (i.e., 30 frames per second of video).

One or more 3D motion sensors 110 may be located within the room of thepatient or individual being monitored 100 and potentially just outsideof the monitored individual's room, home, hospital room, or other placeof temporary or permanent residence. Placing a 3D sensor just outside amonitored individual's room may help the system detect visitors,caregivers, or others who are not the monitored individual a few momentssooner than if there were sensors only inside the room. Detecting othersearlier may help the system track the others and distinguish others fromthe monitored individual without an apparent delay in processing ordisplaying the sensor data, or with a reduced delay in processing ordisplaying the sensor data. The 3D motion sensor is connected to thecomputerized virtual safety zone monitoring system 120 via a dataconnection (USB, TCP/IP, or comparable).

The one or more 3D motion sensors 110 can be configured to recognize themonitored individual 100 and other individuals using biometricidentifiers such as facial tracking, height, distance between points onthe body, etc. Alternately or additionally, the monitored individual 100can be identified by means of a user creating a three-dimensional zonearound the monitored individual 100 through the software application.Once a monitored individual 100 is identified, the software canautomatically generate or allow the user to generate a configurablethree-dimensional virtual safety zone 500 where the caregiver wants toprevent a monitored individual's hand(s) from entering.

Data from the one or more 3D motion sensors 110 are sent to acomputerized virtual safety zone monitoring system 120. The computerizedvirtual safety zone monitoring system 120 (or “computerized monitoringsystem”) is a computer programmed to monitor transmissions of data fromthe 3D motion sensor 110. The computerized monitoring system may beintegral to the 3D motion sensor 110 or a distinctly separate apparatusfrom the 3D motion sensor 110, possibly in a remote location from 3Dmotion sensor 110 provided that the computerized monitoring system 120can receive data from the 3D motion sensor 110. The computerizedmonitoring system 120 may be located in the monitored individual's roomor location. The computerized monitoring system 120 may be connected toa centralized video monitoring station (or “centralized monitoringstation”) 130. The computerized monitoring system 120 and centralizedmonitoring station 130 may be remotely located at any physical locationsso long as a data connection exists (TCP/IP or comparable) between thecomputerized monitoring system 120, the computerized communicationsystem 140 (if separate from computerized monitoring system 120), thecentralized monitoring station 130, and the 3D motion sensor(s) 110.

The computerized communication system 140 is a computer programmed tofacilitate communication between the monitored individual 100 andcomputerized monitoring system 120 in the event the monitored individual100 has placed one or both hands in a virtual safety zone. Thecomputerized communication system 140 may include, but is not limitedto, amplified speakers, microphones, lights, monitors, computerterminals, mobile phones and/or other technologies to allow for theelectronic communication to take place. The computerized communicationsystem may preferably be located within the monitored individual's room;however, certain components of the system are mobile by their nature(i.e., mobile phones, pagers, or certain computers, such as laptop ortablet computers) and could be located at any location so long as a dataconnection (TCP/IP or comparable) exists between the computerizedmonitoring system 120, the computerized communication system 140, thecentralized monitoring station 130, and the 3D motion sensor 110.

The computerized virtual safety zone monitoring system 120 provides thecentralized monitoring station 130 with visual telemetry of themonitored individual 100. This information is received from one or morecomputerized monitoring systems 120, computerized communication systems140 and/or 3D motion sensors 110. The centralized monitoring station 130displays the information in an organized manner to an individual orgroup of individuals assigned to monitor the monitored individuals. Thecentralized monitoring station 130 may be located with or near themonitored individual, e.g., at a nursing station on the same floor as ahospital patient, or may be located remotely from the monitoredindividual. As one example, a computerized virtual safety zonemonitoring system 120 used with a 3D motion sensor 110 in a homeenvironment may be monitored by an agency or individual in a differentpart of a building, a different building, or even a different city. Thecomputerized monitoring system 120 receives the raw data from the camerasensor, processes the raw data, e.g., to determine whether a monitoredindividual's hand or hands have entered a virtual safety zone, andtransmits at least visual telemetry, possibly with sound telemetry,alert information, and/or other data, to the centralized monitoringstation 130.

The centralized monitoring station 130 may comprise a primary display200, as shown in FIG. 2. The centralized monitoring station primarydisplay 200 may be connected to the centralized monitoring station 130,and may show video and/or audio of all locations being monitored at thecentralized monitoring station 130. If audio telemetry is available, itmay be selectable for a particular location, so as to preventoverlapping audio feeds from becoming unintelligible. The centralizedmonitoring station 130 may comprise an alert display 210. Thecentralized monitoring alert display 210 may be a physically separatedisplay from the centralized monitoring primary display 200, e.g., aseparate video display screen or screens. Alternately, or additionally,centralized monitoring alert display 210 may be a portion of centralizedmonitoring primary display 200, or may be presented as a change inappearance, formatting, positioning, or the like in centralizedmonitoring primary display 200.

FIG. 3 shows an exemplary display 300 of visual telemetry data formultiple monitored individuals 100A, 100B, and 100C, in simultaneousviews 310A, 310B, and 310C, respectively, as might be configured oncentralized monitoring station primary display 200. As shown, views310A, 310B, and 310C appear on a split screen, however, different viewscould also be shown on separate displays. In addition to showingmonitored individuals 100A, 100B, and 100C, display 300 shows skeletonFIGS. 320A, 320B, and 320C for each monitored individual. In addition,view 310C shows a pop-up menu 330, which may present configurationoptions for view 310C or options for responding to an alarm associatedwith monitored individual 100C or both.

FIG. 4 shows an exemplary display 300 of visual telemetry data for asingle monitored individual 100, with skeleton FIG. 320 and menu 400.FIG. 5 shows the same exemplary display 300 as FIG. 4 after a user hasselected a menu option to define and/or confirm a virtual safety zone500 and a bed zone 510. The virtual safety zone 500 and/or bed zone 510may be automatically generated by the computerized monitoring system120. For example, computerized monitoring system 120 may define avirtual safety zone 500 around skeleton FIG. 320 by generating aperimeter using a default average distance from key skeleton points,such as face or shoulder landmarks to the perimeter. A bed zone 510 maybe defined by computerized monitoring system 120, for example, bygenerating a perimeter using a default average distance from theperimeter of bed zone 510 to skeleton FIG. 320. While the virtual safetyzone 500 defines an area where the caregiver does not want the monitoredindividual's hand(s), the bed zone 510 may be used to “lock on to” themonitored individual, so that the tracking algorithms do notinadvertently shift to caregivers or visitors if different individuals'body parts cross in the camera view during interactions between themonitored individual and others. Although described as a “bed” zone, bedzone 510 need not be centered on a bed. A bed zone 510 may be definedbased on the monitored individual's skeleton figure, or may be definedaround other furniture or medical equipment supporting the monitoredindividual, such as a chair, chaise longue, surgical table, etc.

The virtual safety zone 500 and/or bed zone 510 may be defined in 2dimensions, e.g., as a perimeter around at least a portion of skeletonFIG. 320. Virtual safety zone 500 may encompass at least a portion ofthe monitored individual's head, neck, shoulders, and/or chest. Virtualsafety zone 500 and bed zone 510 are depicted as rectangular, however,any desired shape could be used, including, without limitation, circles,squares, triangles, ovals, other regular shapes, or irregular shapes. Byselecting a configuration option from menu 400, a user may alter orreset the perimeter that defines virtual safety zone 500.

As shown in FIG. 6, the virtual safety zone 500 may have a thirddimension of depth, e.g., be defined as a volume around at least aportion of skeleton FIG. 320. As with the perimeter of virtual safetyzone 500, the depth of virtual safety zone may be automaticallygenerated by the computerized monitoring system 120. By selecting aconfiguration option from menu 400, a user may alter or reset the depththat defines virtual safety zone 500 using a pop-up menu 330.Alternately, the perimeter and/or depth of virtual safety zone 500 maybe determined entirely by a system user, such as by entering coordinatesor distances, as shown in pop-up menu 330 in FIG. 6, or by providingselection tools like drag-and-drop and pull-to-expand boxes or othershapes. Virtual safety zone 500 may be most often used around the head,neck, shoulders and/or chest. Virtual safety zone 500 could be usedaround other body parts, such as an arm or a leg, however, bedding,dining tables, and other objects may be more likely to obscure the viewof other body parts and cause false alarms or missed alarms.

FIG. 7 shows a configured virtual safety zone 500 overlaid on visualtelemetry for a monitored individual 100. FIG. 8 shows additionalconfiguration options 800 from menu 400, allowing a user to selectwhether to display video telemetry (“VIDEO FEED”), audio telemetry(“AUDIO FEED”), or both.

FIG. 9 shows exemplary configuration settings in a pop-up menu 330. Tab900 presents an exemplary tab for configuring alerts that might beissued if the monitored individual 100 places one or both hands insidevirtual safety zone 500, or places one or both hands inside virtualsafety zone 500 for longer than a predetermined period of time. FIG. 10shows a drop down menu 1000 as may appear upon selection of alerts tab900. A user may be asked to configure an alert for a particular kind ofevent 1010. As shown in FIG. 10, the user has elected to set an alertfor when the camera (e.g., 3D motion sensor 110) is turned off. FIG. 11shows the configuration of an action 1030 from dropdown menu 1000A. Asshown in FIG. 11, the user has selected highlight/flash 1040, a settingthat changes the appearance of the display border, text, or otherproperties on centralized monitoring primary display 200, centralizedmonitoring alert display 210, or both.

FIG. 12 shows an alternative configuration screen in pop up menu 330after selections have been made for event 1010 and action 1030. In theexample of FIG. 12, the appearance of the display border, text, or otherproperties on centralized monitoring primary display 200, centralizedmonitoring alert display 210, or both will change (e.g., be highlightedor flash) if event 1020A occurs. In this case, the display appearance isconfigured to change if the patient enters the virtual safety zone,e.g., if the monitored individual places one or both hands in thevirtual safety zone. The language used in the configuration menus isitself adaptable and configurable, and can be modified for a particularuser group or environment. For example, hospital settings may refer to“patients” and use clinical terminology and/or healthcare industryshorthand. Monitoring services in home environments might refer to“clients” and use lay terminology. A particular institution or usergroup may have the option to configure the menu terminology when thesystem is initially installed, or as system maintenance and/or updatesare undertaken, so that the menu terminology is familiar to theparticular facility or user group adopting the system.

FIG. 13 shows an exemplary configuration for a different alert. Theconfigured event 1020A is still “PATIENT ENTERS SAFETY ZONE”, but action1030 has been configured to “SOUND ALARM” 1040A. On selecting “SOUNDALARM” 1040A, menu item 1300, labeled N/A (e.g., for Notification/Alarm)becomes active for user selection of configurable options. As shown inFIG. 13, the user has selected an alarm from a particular file. Thesound file associated with the alarm might be played audibly on speakersin the room with monitored individual 100 or on a paging or speakersystem in a facility more broadly than the room with monitoredindividual 100 (e.g., audible at a central nursing station in a hospitalunit, or in hallways or break rooms). The sound file may also be sent toa mobile device or other device, e.g., attached to an e-mail or textmessage, or using an autodialing protocol that can deliver a message toa person who answers a phone call or to a voice mail account associatedwith a dialed number. Alternately, or additionally, the sound file maybe played at the central monitoring station 130 to alert an attendantthere.

FIG. 14 shows exemplary pop up menu 330 after an alert has beenconfigured. On selecting alert tab 900, alert 1400, “PATIENT ENTERSSAFETY ZONE: HIGHLIGHT/FLASH” is shown. The description and/or level ofdetail displayed regarding alert 1400 can vary. If additional alertswere configured, they would appear in space 1410, preferably withdescriptions of at least the event and the action for each alert. Inthis way, if multiple alerts are configured, it is easier to find aparticular alert for later editing or deletion, if desired.

With reference to FIGS. 1 and 5, the computerized monitoring system 120determines whether a monitored individual has placed one or both handswithin the virtual safety zone 500 by creating a configurablethree-dimensional zone or perimeter around any locations the caregiverdetermines that the monitored individual should not place their hand(s).The user may also define a predetermined length of time that themonitored individual's hand or hands must be in the virtual safety zone500 before triggering an alarm. The 3D motion sensor 110 can beprogrammed to lock on the monitored individual 100, e.g., using a bedzone 510 as described below, and can send back to the computerizedmonitoring system 120 the 3D coordinates of the joints in theindividual's body and a skeletal outline 320 of the monitored individual100. The system is also able to recognize hand, body, and othermovements and uses the information received from the 3D motion sensor110 to determine if the monitored individual's hand(s) are in thevirtual safety zone 500, how long the hand has been in that zone andwhat, if any, gestures the individual is making.

When the computerized virtual safety zone monitoring system 120 detectsa monitored individual's hand(s) within the virtual safety zone 500,shown as 170 in FIG. 1, the computerized virtual safety monitoringsystem 120 may notify the computerized communication system 140, whichmay send an automated alert to the monitored individual 100 and/orcaregiver(s) 160 that the monitored individual 100 has placed his or herhand(s) in the virtual safety zone 500. The detection may involveinferring the location of the monitored individual's hand(s), forexample, based on the ends of the skeleton figure segments correspondingto the monitored individual's wrists. If facial tracking software isused, a hand or hands lifted to or near the face would obscure thefacial tracking, i.e., one or more of the facial features being used forthe facial tracking would be obscured by the hand or hands. As such, apartially lost or obscured signal could be used to determine that themonitored individual's hand(s) have entered the virtual safety zone 500.An electronic record of the entry into the virtual safety zone 500 maybe made in the database 150. The method of these alerts includes, but isnot limited to, amplified speakers, microphones, lights, monitors,computer terminals, mobile phones, and/or other technologies to allowfor the electronic communication to take place. The alert to themonitored individual 100 may include an audible or visual instruction,such as “please put your hands down,” “please stop what you are doing,”or “please wait, help is coming.” For example, a paging system may beused to make an oral announcement in the monitored individual's room, ora text message may be displayed on a television or computer display inthe monitored individual's room. The purpose of the alert may be toinstruct the patient to discontinue activity consistent with disturbingmedical equipment and/or temporary implants. The alerts may be availablein two or more languages, allowing the system to provide instructionseven if the monitored individual 100 does not speak the same primarylanguage as the monitored individual's caregivers 160. The alert may beconfigured, using menus like those described above, to specify whatlanguage should be used for alerts to the monitored individual 100.

When an alert is triggered, the alert may be sent, at least initially,to the monitored individual 100, to give monitored individual 100 anopportunity to self-correct before alerting the central monitoringstation 130 and/or caregivers 160. Alternately, central monitoringstation 130 may be alerted with or even before the monitored individual,so that central monitoring station 130 can determine whether the entryinto the virtual safety zone 500 is, in fact, problematic. The centralmonitoring station, or an attendant there, can do this by viewing thelive video and/or audio feed from the 3D motion sensor(s), anddetermining whether the gestures or motions appear to be dangerous. Thecentral monitoring station 130 attendant could then approve alert(s) toappropriate caregiver(s) 160 to intervene. In another alternative, oneor more caregiver(s) 160 local to the monitored individual 100 can bealerted with or even before the monitored individual 100, so that thecaregiver(s) 160 can assess what is happening in person. Or, themonitored individual 100, caregiver(s) 160, and the central monitoringstation 130 could all be alerted at the same time. The priority andtiming of alerts to different individuals or stations can be configuredin accordance with the needs and desires of a particular facility,experience with a particular monitored individual or type of patient, orany other criterion of the system owner or user. This is true forinitial alerts as well as continuing alerts (e.g., if a monitoredindividual receives an audible alert and does not remove his or herhands from the virtual safety zone) or repeated alerts (two or moredistinct events where the monitored individual's hand or hands enter thevirtual safety zone). The priority and timing of alerts to differentindividuals may be different for initial, continuing, and/or repeatedalerts.

At step 180 in FIG. 1, the computerized virtual safety zone monitoringsystem 120 determines whether the monitored individual 100 has removedhis or her hand(s) from the virtual safety zone 500. If the systemdetects that a monitored individual 100 has removed his or her hand(s)from the virtual safety zone 500, an electronic record can be made indatabase 150 and audible and/or visible alerts may be issued to themonitored individual 100 and/or caregiver(s) 160 to confirm compliance.The computerized virtual safety zone monitoring system 120 may continueto analyze incoming data from 3D motion sensor 110 for continuedcompliance.

Should the monitored individual 100 fail to remove his or her hand(s)from the virtual safety zone 500, an audible and/or visible alert can begiven to the monitored individual 100 and/or caregiver(s) 160, notifyingthe monitored individual 100 and caregiver(s) 160 that the monitoredindividual 100 or caregiver 160 needs to take measures to remove themonitored individual's hand(s) from the virtual safety zone 500. Thealert can be sent by the computerized communication system, which caninclude, but is not limited to, a system of speakers, microphoneslights, monitors, mobile phones, and methods of communication includingbut not limited to voice, email, SMS messaging, video, phone calls, orflashing lights. A second or subsequent alert to the monitoredindividual 100 maybe worded more strongly than the initial alert, usinglanguage like “Stop now. Help is coming.” The computerized monitoringsystem 120 may monitor, using gesture recognition, location tracking,facial tracking, skeleton figure tracking, or other measures, whetherthe monitored individual 100 has taken appropriate steps to remove hisor her hand(s) from the virtual safety zone 500 based on the informationreceived from the 3D motion sensor 110. Facial recognition could also beused, however, facial tracking is typically sufficient for the purposeof monitoring a virtual safety zone. An electronic record can be made inthe database 150 and additional audible and/or visible alerts can beissued to the monitored individual 100 and/or caregiver(s) 160 until themonitored individual 100 removes his or her hand(s) from the virtualsafety zone 500. Captured video can also be stored and/or reviewed bythe computerized monitoring system 120 when the system makes itsdetermination.

If the monitored individual 100 places his or her hand(s) in the virtualsafety zone 500, notification may be given to the caregivers 160 orother designated persons. Notification of caregivers can be made throughphone call, text messaging, speakerphone systems, email, or otherelectronic means of communication. The system database 150 may also beupdated to reflect actions taken.

FIG. 2 shows the workflow for centralized monitoring and alertingregarding whether a monitored individual's hand(s) have encroached intoa virtual safety zone through the use of 3D motion sensors 110. One ormore 3D motion sensors, shown as 110A, 110B, and 110C in FIG. 2, areinstalled in and/or just outside a monitored individual's room, home,hospital room, or other place of temporary or permanent residence andconnected to computerized monitoring and communication systems as shownin FIG. 1. Video and/or audio data for a monitored individual, such asindividuals 100A, 100B, and 100C, is collected by 3D motion sensors110A, 110B, and 110C, respectively. The video and/or audio data istransferred by the 3D motion sensors 110A, 110B, and 110C tocomputerized monitoring and communication systems 120A, 120B, and 120C,respectively. Video, audio, and alert data is sent by the computerizedmonitoring and communication systems 120A, 120B, and 120C to acentralized monitoring station 130 where the data is aggregated forvarious monitored individuals. The computerized monitoring systems 120A,120B, and 120C receive the raw data from the 3D motion sensors 110A,110B, and 110C, run the skeletal recognition, facial tracking, and/orgesture recognition algorithms, and then send the audio, video, andalert data to the centralized monitoring station 130. The centralizedmonitoring station 130 receives and displays this data from one or moresensors/computerized monitoring systems. Similar to a grid of camerasbeing watched on a screen (i.e., where a plurality of camera feeds areviewed on a single screen), the centralized monitoring stationaggregates the various video feeds, as it receives and displaysinformation from multiple cameras. Preferably, the centralizedmonitoring station 130 receives data at all times, e.g., continuously,from the 3D motion sensors 110A, 110B, and 110C, typically viacomputerized monitoring and communication systems 120A, 120B, and 120C,to allow the various individuals to be constantly monitored at thecentralized monitoring station 130 regardless of whether or not a breachof the virtual safety zone 500 has been detected.

All video, audio, and/or alert feeds received by the centralizedmonitoring station 130 can be displayed on the centralized monitoringprimary display 200. Alternatively, multiple centralized monitoringprimary displays can be utilized based on the quantity of rooms to bemonitored at a given time.

When the centralized monitoring station 130 receives an alert from anyof the computerized monitoring and communication systems 120A, 120B,120C, indicating that a monitored individual 100 has placed his or herhand(s) in the virtual safety zone 500, the video, audio, and/or alertinformation for that particular individual is displayed on thecentralized monitoring alert display 210. An alert can be represented inone or more different types of physical configurations. It can be avisual queue on screen at the centralized monitoring station 130 such asthe specific camera view flashing or being highlighted in a color todraw attention to that display among others. It can be an audible sound(voice or alarm type sound) at the centralized monitoring station 130,an audible sound at the computerized monitoring system 120 attached tothe 3D motion sensor 110, a text message, an email, turning on a light,or even running a program on a computer. Should the centralizedmonitoring station 130 receive alerts from more than one of thecomputerized monitoring and communication systems 120A, 120B, 120C,indicating that a monitored individual 100 has placed his or her hand(s)in the virtual safety zone 500, the centralized monitoring alert display210 will display the video, audio, and/or alerting information from allsuch instances at the same time. If no alert is received by thecentralized monitoring station 130, nothing is displayed on thecentralized monitoring alert display 210. Preferably, all monitoredindividual rooms can be displayed and visible on the central monitoringprimary display 200 whether alerting or not. When an alert is generated,attention can be drawn to the particular camera and a duplicativedisplay of the alerting camera can be displayed on a second separatecomputer monitor, e.g., the centralized monitoring alert display 210.

An electronic record of any alerts received by the centralizedmonitoring station 130 can be stored in a database 150.

The various components described can be in electrical, wired, and/orwireless communication with each other. The various computerized systemsand processors as described herein may include, individually orcollectively, and without limitation, a processing unit, internal systemmemory, and a suitable system bus for coupling various systemcomponents, including database 150, with a control server. Computerizedmonitoring system 120 and/or centralized monitoring station 130 mayprovide control server structure. The system bus may be any of severaltypes of bus structures, including a memory bus or memory controller, aperipheral bus, and a local bus, using any of a variety of busarchitectures. By way of example, and not limitation, such architecturesinclude Industry Standard Architecture (ISA) bus, Micro ChannelArchitecture (MCA) bus, Enhanced ISA (EISA) bus, Video ElectronicStandards Association (VESA) local bus, and Peripheral ComponentInterconnect (PCI) bus.

The computerized systems typically include therein, or have access to, avariety of computer-readable media, for instance, database 150.Computer-readable media can be any available media that may be accessedby the computerized system, and includes volatile and nonvolatile media,as well as removable and non-removable media. By way of example, and notlimitation, computer-readable media may include computer-storage mediaand communication media. Computer-storage media may include, withoutlimitation, volatile and nonvolatile media, as well as removable andnon-removable media implemented in any method or technology for storageof information, such as computer readable instructions, data structures,program modules, or other data. In this regard, computer-storage mediamay include, but is not limited to, RAM, ROM, EEPROM, flash memory orother memory technology, CD-ROM, digital versatile disks (DVDs) or otheroptical disk storage, magnetic cassettes, magnetic tape, magnetic diskstorage, or other magnetic storage device, or any other medium which canbe used to store the desired information and which may be accessed byone or more of the computerized systems. Computer-storage media excludessignals per se.

Communication media typically embodies computer readable instructions,data structures, program modules, or other data in a modulated datasignal, such as a carrier wave or other transport mechanism, and mayinclude any information delivery media. As used herein, the term“modulated data signal” refers to a signal that has one or more of itsattributes set or changed in such a manner as to encode information inthe signal. By way of example, and not limitation, communication mediaincludes wired media such as a wired network or direct-wired connection,and wireless media such as acoustic, RF, infrared, and other wirelessmedia. Combinations of any of the above also may be included within thescope of computer-readable media. The computer-storage media discussedabove, including database 150, provides storage of computer readableinstructions, data structures, program modules, and other data for thecomputerized systems.

The computerized systems may operate in a computer network using logicalconnections to one or more remote computers. Remote computers may belocated at a variety of locations, for example, but not limited to,hospitals and other inpatient settings, veterinary environments,ambulatory settings, medical billing and financial offices, hospitaladministration settings, home health care environments, payer offices(e.g., insurance companies), home health care agencies, clinicians'offices and the clinician's home, the patient's own home, or over theInternet. Clinicians may include, but are not limited to, a treatingphysician or physicians, specialists such as surgeons, radiologists,cardiologists, and oncologists, emergency medical technicians,physicians' assistants, nurse practitioners, nurses, nurses' aides,pharmacists, dieticians, microbiologists, laboratory experts, laboratorytechnologists, genetic counselors, researchers, veterinarians, students,and the like. The remote computers may also be physically located innon-traditional medical care environments so that the entire health carecommunity may be capable of integration on the network. The remotecomputers may be personal computers, servers, routers, network PCs, peerdevices, other common network nodes, or the like, and may include someor all of the elements described above in relation to the controlserver. The devices can be personal digital assistants or other likedevices.

Exemplary computer networks may include, without limitation, local areanetworks (LANs) and/or wide area networks (WANs). Such networkingenvironments are commonplace in offices, enterprise-wide computernetworks, intranets, and the Internet. When utilized in a WAN networkingenvironment, the control server may include a modem or other means forestablishing communications over the WAN, such as the Internet. In anetworked environment, program modules or portions thereof may be storedin the control server, in the database 150, or on any of the remotecomputers. For example, and not by way of limitation, variousapplication programs may reside on the memory associated with any one ormore of the remote computers. It will be appreciated by those ofordinary skill in the art that the network connections shown areexemplary and other means of establishing a communications link betweenthe computers may be utilized.

In operation, a user may enter commands and information into thecomputerized system(s) using input devices, such as a keyboard, apointing device (commonly referred to as a mouse), a trackball, or atouch pad. Other input devices may include, without limitation,microphones, satellite dishes, scanners, or the like. In addition to orin lieu of a monitor, the computerized systems may include otherperipheral output devices, such as speakers and a printer.

Many other internal components of the computerized system hardware arenot shown because such components and their interconnection are wellknown. Accordingly, additional details concerning the internalconstruction of the computers that make up the computerized systems arenot further disclosed herein.

Methods and systems of embodiments of the present disclosure may beimplemented in a WINDOWS or LINUX operating system, operating inconjunction with an Internet-based delivery system, however, one ofordinary skill in the art will recognize that the described methods andsystems can be implemented in any system suitable for supporting thedisclosed processing and communications. As contemplated by the languageabove, the methods and systems of embodiments of the present inventionmay also be implemented on a stand-alone desktop, personal computer,cellular phone, smart phone, tablet computer, PDA, or any othercomputing device used in a healthcare environment or any of a number ofother locations.

Example 1—Configuring Zones

The bed zone 510 and virtual safety zone 500 for a given 3D motionsensor 110 can be configured. To begin, the user hovers over the 3Dmotion sensor video window with the cursor, then right-clicks, selectsplug-ins, and then selects configure plug-ins. A window will pop upshowing the 3D motion sensors' feed.

The user selects the icon(s) for the type of zone they wish to draw. Inthis non-limiting example, a bed zone 510 and a virtual safety zone 500can be selected.

As non-limiting examples, the following icons can appear on the screenfor selection:

Safety Zone

Bed Zone

Auto Bed Zone (Select Patient)

Auto Bed Zone (Auto-select)

Saved Zones

Clear All

To place a zone, the user clicks on the screen where the user would liketo start the zone. Then, the cursor is moved to the corner point andclicked again. The user continues selecting additional points until thezone is drawn to the user's satisfaction. Preferably, the user clicks onthe round end point of the beginning of the zone to complete the zone.Upon completion the zone appears and has a depth range box preferably inthe middle.

The user can adjust the depth range for any given zone. By doubleclicking on the depth range box, or by using another user selectionmethod, an Edit Depth window can appear. The user can enter in the depthranges (for example, in millimeters (mm)), and then the user can clickthe Save button or icon when the user is done to store the enteredvalues. Although this example uses millimeters for depth range values,depth ranges could be entered in any desired unit of measurementincluding, but not limited to, centimeters, meters, inches, feet, andyards.

If there are any other types of zones to draw for the particular sensor,the above steps can be repeated to place the next zone, and the depthsetting can be adjusted for each if necessary or desired. Additionally,all zones can be cleared by clicking on or otherwise selecting the ClearAll icon preferably located in the toolbar.

Once all zones/wires are configured, the window can be closed to finishor an option to save the zone configuration for later use can beprovided and selected.

To access the main settings window, the user can click on the Settingsmenu and select the Main Settings listing from the drop-down list.Alternately, the user can click on the Gear icon in the toolbar toaccess the main settings window or utilize a designated keyboardshortcut.

Example 2—Configuring an Alert

For one non-limiting way to configure a new Alert, the user can selectan Alerts tab and click on or otherwise select the Add button, which canresult in the Configure Alert box to appear on the screen.

Under the Event field in the Configure box, the user can select theevent from the drop down list that they wish to alert on.

Once the Event type is selected, under the Action field, the user canselect the Action he or she wishes to have the system perform when theselected Event is detected.

For certain Actions an additional field may need to be completed tofinish the Action. If the field is required, it can appear below theAction dropdown. If no further fields are required, the Configure Alertbox can display “N/A” or appear blank.

Once all of the settings are selected, the user clicks on or otherwiseselects the OK button (or similar function button) which saves the newAlert.

The new Alert can now be listed in the Alerts tab window. To edit anexisting Alert, the user first clicks on or otherwise selects the Alertand then selects the Edit button. To delete an Alert, the user can firsthighlight the Alert and then click on the Delete button.

To add more Alerts, the user clicks on or selects the Add button andrepeats steps 4-6. When finished, the user clicks on or otherwiseselects the bottom corner OK button to save and close the window.

Automatically detecting and providing alerts to a monitored individualand/or caregivers when a monitored individual enters a virtual safetyzone may reduce incidences of HAIs by lowering the risk of improperdisturbance and/or removal of medical equipment; increase the survivalrate for individuals who are susceptible to HAIs; reduce costs forhospitalization and medical care related to HAIs; reduce costs forhospitalization and medical care related to re-insertion of improperlyremoved medical equipment; or reduce injury and deaths of monitoredindividuals who have improperly removed medical equipment.

From the foregoing, it will be seen that this disclosure is one welladapted to attain all the ends and objects hereinabove set forthtogether with other advantages which are obvious and which are inherentto the structure.

It will be understood that certain features and subcombinations are ofutility and may be employed without reference to other features andsubcombinations. This is contemplated by and is within the scope of theclaims.

Since many possible embodiments may be made of the invention withoutdeparting from the scope thereof, it is to be understood that all matterherein set forth or shown in the accompanying drawings is to beinterpreted as illustrative and not in a limiting sense.

What is claimed is:
 1. A computerized method for detecting when amonitored individual has moved one or both of his or her hands within avirtual safety zone, the method executed by one or more processors andcomprising: receiving video data from one or more 3D motion sensorscapturing live video data of an individual to be monitored within amonitored area; defining a virtual safety zone representing an area inwhich the individual should not place his or her hands, the virtualsafety zone encompassing at least part of the individual's face;determining that one or more hands of the individual are within thevirtual safety zone based on at least a partially lost or obscuredfacial tracking signal; and upon determining that the one or more handsof the individual are within the virtual safety zone, electronicallytransmitting an alert to at least one designated recipient.
 2. Thecomputerized method of claim 1 further comprising determining whetherone or more hands of the individual remain within the virtual safetyzone for at least a predetermined period of time.
 3. The computerizedmethod of claim 2, wherein the alert is electronically transmitted onlyupon determining one or more hands of the individual remained within thevirtual safety zone for at least the predetermined period of time. 4.The computerized method of claim 2 further comprising continuouslydisplaying a live video feed of the virtual safety zone received fromthe one or more 3D motion sensors on a centralized monitoring alertdisplay after it has been determined that one or more hands of theindividual remained within the virtual safety zone for at least thepredetermined period of time.
 5. The computerized method of claim 1further comprising continuously displaying a live video feed of thevirtual safety zone received from the one or more 3D motion sensors on acentralized monitoring primary display that is remotely located from themonitored area.
 6. The computerized method of claim 5 further comprisingcontinuously displaying a live video feed of the monitored area receivedfrom the one or more 3D motion sensors on a centralized monitoring alertdisplay after it has been determined that one or more hands of theindividual are within the virtual safety zone, wherein the centralizedmonitoring alert display is a separate display from the centralizedmonitoring primary display.
 7. The computerized method of claim 1further comprising updating a database regarding the determination thatone or more hands of the individual was within the virtual safety zone.8. The computerized method of claim 1, wherein the designated recipientincludes one or more of the individual, a caregiver, an alternatecaregiver, and a supervisor.
 9. The method of claim 1, wherein thevirtual safety zone is defined using at least facial tracking torecognize the individual's face.
 10. A system for determining whether amonitored individual has placed one or both hands in a virtual safetyzone, the system comprising: one or more 3D motion sensors capturinglive video data of an individual to be monitored; a computerizedmonitoring system, the computerized monitoring system configured to:receive video data from the one or more 3D motion sensors; define avirtual safety zone representing an area in which the individual shouldnot place his or her hands, the virtual safety zone encompassing atleast part of the individual's face; determine that one or more hands ofthe individual are within the virtual safety zone based on at least apartially lost or obscured facial tracking signal; and upon determiningthat the one or more hands of the individual are within the virtualsafety zone, electronically transmit an alert to at least one designatedrecipient.
 11. The system of claim 10 further comprising: a computerizedcommunication system configured to receive from the computerizedmonitoring system a determination that one or more hands of theindividual entered the virtual safety zone.
 12. The system of claim 11,wherein the computerized monitoring system is further configured toactuate a timer upon determining that one or more hands of theindividual entered a virtual safety zone.
 13. The system of claim 12,wherein the computerized monitoring system sends to the computerizedcommunication system the determination that one or more hands of theindividual entered a virtual safety zone only if the one or more handsremain in the virtual safety zone for at least a predetermined period oftime.
 14. The system of claim 10, wherein the designated recipientincludes one or more of the individual, a caregiver, an alternatecaregiver, and a supervisor.
 15. The system of claim 10, wherein thealert comprises an audible instruction to the monitored individual. 16.The system of claim 10 further comprising a database for logging eventsrelated to the one or more hands of the individual entering the virtualsafety zone.
 17. The system of claim 10, wherein the virtual safety zoneis defined using at least facial tracking to recognize the individual'sface.
 18. Non-transitory computer-readable storage media having storedthereon executable instructions that, when executed by a computer, causethe computer to: receive video data from one or more 3D motion sensorscapturing live video data of an individual to be monitored; define avirtual safety zone representing an area in which the individual shouldnot place his or her hands, the virtual safety zone encompassing atleast part of the individual's face; determine that one or more hands ofthe individual are within the virtual safety zone based on at least apartially lost or obscured facial tracking signal; and upon determiningthat the one or more hands of the individual are within the virtualsafety zone, electronically transmit an alert to at least one designatedrecipient.
 19. The media of claim 18, wherein the virtual safety zone isdefined using at least user input.
 20. The media of claim 18, whereinthe instructions further cause the computer to log the alert in adatabase.